Skip to content

Use PiecewiseSequence for List take elements#8833

Merged
robert3005 merged 7 commits into
developfrom
codex/list-take-piecewise-elements
Jul 21, 2026
Merged

Use PiecewiseSequence for List take elements#8833
robert3005 merged 7 commits into
developfrom
codex/list-take-piecewise-elements

Conversation

@danking

@danking danking commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This is just a simplification. List::take already had a specialized implementation. The new PiecewiseSequence specializations in various primitive arrays supersede this change.

EDIT: Heh, I guess the PiecewiseSequence stuff is actually even faster than old List::take. Nice.

codex:

Summary

  • replace List take's dense per-element index expansion with PiecewiseSequence element runs
  • preserve FSL-style null handling: null index/source rows emit zero-length placeholder runs while output validity still comes from Validity::take
  • let the elements child choose its own optimized take implementation through the normal ArrayRef::take path

Validation

  • cargo +nightly fmt --all
  • cargo test -p vortex-array arrays::list::compute::take
  • cargo check -p vortex-array
  • cargo clippy -p vortex-array --all-targets --all-features -- -D warnings
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will improve performance by 29.27%

⚡ 10 improved benchmarks
✅ 1778 untouched benchmarks
⏩ 54 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation take_filter_list_nullable_random_mask_random_indices[256, 50] 130.3 µs 99.4 µs +31.11%
⚡ Simulation take_filter_list_slice_mask_random_indices[768, 50] 121 µs 92.5 µs +30.83%
⚡ Simulation take_filter_list_slice_mask_random_indices[256, 50] 120.3 µs 92.2 µs +30.45%
⚡ Simulation take_filter_list_nullable_random_mask_random_indices[768, 50] 134.4 µs 103.7 µs +29.68%
⚡ Simulation take_filter_list_random_mask_random_indices[256, 50] 124.1 µs 95.7 µs +29.61%
⚡ Simulation take_filter_list_small_random_mask_random_indices[256, 10] 117.1 µs 90.7 µs +29.08%
⚡ Simulation take_filter_list_small_uncached_random_mask_random_indices[256, 10] 118 µs 91.7 µs +28.66%
⚡ Simulation take_filter_list_small_random_mask_random_indices[768, 10] 121.3 µs 94.9 µs +27.86%
⚡ Simulation take_filter_list_small_uncached_random_mask_random_indices[768, 10] 121.3 µs 95 µs +27.76%
⚡ Simulation take_filter_list_random_mask_random_indices[768, 50] 128 µs 100.3 µs +27.71%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

Comparing codex/list-take-piecewise-elements (a9f9ecd) with develop (ae9ecec)

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from c4a0358 to c76531d Compare July 17, 2026 20:05
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from 77d37d8 to b7ad4df Compare July 17, 2026 20:15
@danking danking added the changelog/performance A performance improvement label Jul 17, 2026
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 80f7b89 to 30870aa Compare July 17, 2026 20:19
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from b7ad4df to f0eec2d Compare July 17, 2026 20:19
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 30870aa to e777ab4 Compare July 17, 2026 20:25
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from f0eec2d to d40cee8 Compare July 17, 2026 20:25
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from e777ab4 to e4a0543 Compare July 17, 2026 20:37
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from 243ed19 to a237158 Compare July 20, 2026 16:11
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 490e9db to ed7bbe2 Compare July 20, 2026 16:11
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from a237158 to e134f5b Compare July 20, 2026 16:37
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch 2 times, most recently from efafe71 to 56b0394 Compare July 20, 2026 16:56
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 2 times, most recently from ffa4184 to e88d96f Compare July 20, 2026 18:10
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 56b0394 to 80bfa43 Compare July 20, 2026 18:10
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from e88d96f to 3d3a91c Compare July 20, 2026 19:24
@danking
danking force-pushed the codex/piecewise-sequential-run-take-consumers branch from 80bfa43 to 09ca199 Compare July 20, 2026 19:24
@danking
danking force-pushed the codex/list-take-piecewise-elements branch 5 times, most recently from 655fad8 to e05bf6e Compare July 20, 2026 20:17
robert3005
robert3005 reviewed Jul 20, 2026
View reviewed changes
Comment thread vortex-array/src/arrays/list/compute/take.rs Outdated
robert3005
robert3005 reviewed Jul 20, 2026
View reviewed changes
Comment thread vortex-array/src/arrays/list/compute/take.rs
Base automatically changed from codex/piecewise-sequential-run-take-consumers to develop July 20, 2026 22:00
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from e05bf6e to e0096ca Compare July 20, 2026 22:04
@github-actions

github-actions Bot commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done a9f9ecd 1 Explore Profiling Data
Previous Runs (2)
Status Commit Job Attempt Link
🟢 Done 75347f8 1 Explore Profiling Data
🟢 Done e0096ca 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.9%
Engines: DataFusion No clear signal (+5.4%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 1.014x ➖
Parquet (geomean): 1.004x ➖
Shifts: Parquet (control) +0.4% · Median polish +0.4%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
datafusion / vortex-file-compressed (1.056x ➖, 0↑ 0↓)
name PR a9f9ecd (ns) base 994591b (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10043212 9478149 1.06
vortex_q01/datafusion:vortex-file-compressed 6394650 6075184 1.05
datafusion / parquet (1.002x ➖, 0↑ 0↓)
name PR a9f9ecd (ns) base 994591b (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20329335 20755706 0.98
vortex_q01/datafusion:parquet 4690538 4576415 1.02
duckdb / vortex-file-compressed (0.991x ➖, 0↑ 0↓)
name PR a9f9ecd (ns) base 994591b (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10054001 10237816 0.98
vortex_q01/duckdb:vortex-file-compressed 6328473 6322510 1.00
duckdb / parquet (1.007x ➖, 0↑ 0↓)
name PR a9f9ecd (ns) base 994591b (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23665065 23520974 1.01
vortex_q01/duckdb:parquet 9535058 9461751 1.01

No file size changes detected.

@robert3005
robert3005 marked this pull request as ready for review July 21, 2026 09:19
danking added 6 commits July 21, 2026 11:00
@danking
Use PiecewiseSequence for List take elements
4480586 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Cover null List take placeholders
4c52cab 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Normalize List take indices before range rebuild
a379c69 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Avoid normalizing unused null list indices
91866d8 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Rely on List offset invariants in List take
a4326d1 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Use valid null List ranges in take test
75347f8 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking force-pushed the codex/list-take-piecewise-elements branch from e0096ca to 75347f8 Compare July 21, 2026 15:00
@danking
Shorten list take output offset type name
a9f9ecd 
Signed-off-by: Daniel King <dan@spiraldb.com>
@robert3005
robert3005 merged commit 1e2df53 into develop Jul 21, 2026
83 checks passed
@robert3005
robert3005 deleted the codex/list-take-piecewise-elements branch July 21, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robert3005 robert3005 robert3005 approved these changes

Assignees

No one assigned

Labels

changelog/performance A performance improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@danking @robert3005